Repository navigation
Run the command-line tests three at a time, and check the oracles' mutation score on every change - #391
Merged
Conversation
… of the clock A run can now be told to stop once it has sent so many requests: two settings in the schedule group, `stopAfterRequests` (off) and `requestsBeforeStopping` (1000). On, the loop stops asking for more at that count and waits for the answers it is owed, as it does at a deadline; the run ends as it was meant to, not cut short. A run of a known size sends the same requests on a slow machine as on a fast one, which is what RESTest's own tests of the command line needed: 65 of their runs were given a second or two of the clock, and twice this week the slowest Windows machine continuous integration runs on spent that second starting up and answered 3, "nothing was tested" (AuthCommandTest, on master). Fifteen test classes now give their runs a size; the module takes 1 min 20 s instead of 3 min. Two constants of the command move to the same group, by the rule of ADR-0025: how long a run stopped from outside may spend writing what it found once it has stopped waiting for its answers (`interruptWritingGrace`, 5 s) and how long it then gives its directory to say what was written (`interruptLookingGrace`, 1 s). ARunCutShortTest sets the first to a second and the second to ten: the test took eight seconds and failed on a slow Windows machine whose directory did not answer within the one second it was given (the other red run on master this week); it now takes one second and does not depend on the machine. The settings page, the switches page (a second switch that is not a lever) and the command-line page say so; ADR-0025 records the four settings. RunLoopTest proves a run of twelve requests sends twelve, waits for their answers and is not cut short. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ons on one row, and stop watching v2 Three changes to how a build is checked, none to what is checked. The command-line module's tests run in three processes instead of one (`forkCount` in its POM). They are the command run end to end, each class against a stand-in API on a port of its own and writing under a directory of its own, and most of their time is waiting for an API to answer; the module took three of the four minutes of `./mvnw verify`, and takes a third of that now. Nothing a run keeps is shared between two processes, which the architecture rules' ban on static mutable state is there to guarantee. The mutation score of the oracles is checked on every change, on the `ubuntu-latest` / Java 25 row, as a step after the build. `docs/ci.md` said it "costs minutes" and so was left to be run by hand, which nobody did unless a pull request touched the oracles; it takes eleven seconds (176 mutations, 97% killed). A `mutation` profile in `restest-oracles/pom.xml` binds the score to that module's `verify`, so the ordinary build still does not pay for it. `ci.yml` no longer runs for `v2`, a branch no commit has reached since 3 October and that is seventy-six behind `master`. The CI page says what runs now, names the five rules its table of gates had left out, and loses four sentences from the weeks when the modules were empty. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Wrapping only: the arguments of the runs given a size, three constructions of the schedule settings in tests, one test's name and one paragraph of Javadoc. Nothing a test does changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
11 tasks done
…ed to run three at a time With three test processes on a runner of four cores, the tests that gave a run one to three seconds of the clock failed on Linux and Windows: the machine was busy, the second went on starting up, and the run sent nothing. #390 gives those runs a number of requests instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ssegura
enabled auto-merge (squash)
October 14, 2026 17:13
ssegura
added a commit
that referenced
this pull request
Oct 14, 2026
**Related:** None ## What you can do now that you could not before A release no longer carries an empty `restest-idl` jar, and the repository no longer lists a module for the IDL language, inter-parameter dependencies, that nothing reads yet. The known limitations now say plainly that dependencies between parameters are not read, which a user of RESTest 1.x will look for. ## Try it yourself ```bash git fetch && git switch chore/idl-module-out-of-the-reactor ./mvnw -q package -DskipTests dist/assemble.sh target/dist > /dev/null && ls target/dist/restest-*/lib | grep '^restest-' ``` Real output: ``` restest-cli-2.1.1-SNAPSHOT.jar restest-core-2.1.1-SNAPSHOT.jar restest-exec-2.1.1-SNAPSHOT.jar restest-gen-2.1.1-SNAPSHOT.jar restest-oracles-2.1.1-SNAPSHOT.jar restest-report-2.1.1-SNAPSHOT.jar restest-spec-2.1.1-SNAPSHOT.jar restest-store-2.1.1-SNAPSHOT.jar ``` On `master` the list also has `restest-idl-2.1.1-SNAPSHOT.jar`, whose only class is `module-info.class`. ## How it works RESTest is built from modules that may only depend inwards, towards `restest-core`. One of them, `restest-idl`, was created at the start to hold the IDL language and a constraint solver, and has held nothing else since. It was still compiled, packaged, copied into every release and listed among what the tool is made of. It leaves the build until the first code that reads IDL arrives. The rule that checks the dependencies between modules keeps a place for it, so it comes back exactly where ADR-0004 put it, checked from its first class. ## What changed in the code - Removed: `restest-idl/` and the dependencies of `restest-gen`, `restest-cli` and `restest-arch-tests` on it; the two `requires io.restest.idl` in `module-info.java`. - `ArchitectureRules`: unchanged, with a comment on why the `Idl` layer stays. - `AGENTS.md`, `docs/DESIGN.md`, `docs/ci.md`, the root and arch-tests `pom.xml` comments: eight production modules. - ADR-0004 and ADR-0010 amended, and the index; `docs/known-limitations.md` says dependencies between parameters are not read. ## Evidence - [x] Tests: none added; `./mvnw verify` green locally. CI: this pull request's checks. - [x] Architecture tests: unchanged, none weakened; the layered rule keeps its `Idl` layer. - [x] Mutation score: not applicable. - [x] Per-request overhead: not applicable. - [x] Smoke run, two containerised APIs: the smoke job of this pull request. - [x] Idle time: not applicable. - [x] Artefact: the list above. ## Decisions taken - **The module leaves; the rule's layer stays.** Removing the layer would have been a change to an architecture test for no gain, and keeping it means nothing has to be decided again when IDL arrives. - **`AGENTS.md` changes** because it lists the modules; ADR-0030 says such a change is made in a pull request, and this is one. - **Order of merging.** #391, which runs the command-line tests three at a time, edits the same paragraph of `docs/ci.md`, and #397, which moves to 2.2.0-SNAPSHOT, changes `restest-idl/pom.xml`; whichever is merged second keeps the eight modules and deletes the file. ## Open questions None. --- - [x] Targets `master` - [x] English throughout, including comments and test names - [x] The hard rules in `AGENTS.md` hold - [x] Nothing from "Out of scope" in `docs/DESIGN.md` without explicit approval 🤖 Generated with [Claude Code](https://claude.com/claude-code)
…kes a couple of minutes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related: None. Builds on #390.
What you can do now that you could not before
Nothing a user sees changes. A contributor gets an answer from CI sooner - the command-line tests run
three classes at a time - and a pull request that weakens the oracles' tests now fails CI, because
the mutation score of
restest-oraclesis checked on every change rather than only when somebodyremembers to run it.
Try it yourself
The mutation score, as one row of CI now runs it:
git fetch && git switch ci/faster-rows ./mvnw --batch-mode --no-transfer-progress -Pmutation verify -pl restest-oracles -am -Djacoco.skip=trueReal output, the lines that matter, in 15 seconds on the maintainer's Mac:
The command-line tests, now three classes at a time:
restest-cli's tests, on the maintainer's MacHow it works
Most of the build's time is the command-line tests, which run the whole tool against a stand-in API
and spend their time waiting for it to answer. Each test class has a stand-in of its own on a port of
its own and writes under a directory of its own, and RESTest keeps no global state, so three classes
can run side by side in three processes without touching each other.
Mutation testing changes the oracles' own code on purpose - a comparison flipped, a call removed - and
checks that the tests notice.
docs/ci.mdsaid it "costs minutes" and left it to be run by hand; ittakes about ten seconds, so the Linux / Java 25 row now runs it after the build, through a
mutationprofile that leaves the ordinary
./mvnw verifyas it was.What changed in the code
restest-cli/pom.xml:forkCount3,reuseForkstrue, with the reason.restest-oracles/pom.xml: amutationprofile binding PIT'smutationCoveragetoverify; thecomment says it is run on one row of CI.
.github/workflows/ci.yml: a step after the build on theubuntu-latest/ Java 25 row; the triggersno longer name
v2, a branch no commit has reached since 3 October and 76 commits behindmaster.docs/ci.md: the triggers, the mutation score as a gate, the five production rules its table ofgates did not name, and four sentences written when the modules were still empty.
CONTRIBUTING.md: the sentence that said the mutation threshold is run on demand.Evidence
./mvnw verifygreen locally. CI: this pull request's checks, including thenew mutation step on the Linux / Java 25 row.
Decisions taken
time. The gain on CI will be smaller than on the Mac, and is not measured yet.
masteras itis, failed on Linux and Windows: with three processes on four cores, the tests that gave a run one
to three seconds of the clock spent them starting up and sent nothing (
CheckSwitchesCommandTest,AuthCommandTest,ABrokenRunAnswersFourTest,AKeyIsWrittenNowhereTest). Stop a run after a set number of requests, and test the command line by size rather than by the clock #390 gives those runs anumber of requests instead, so this branch brought Stop a run after a set number of requests, and test the command line by size rather than by the clock #390's branch in; Stop a run after a set number of requests, and test the command line by size rather than by the clock #390 is merged now, and
masterhas been merged into this branch since, so its diff shows only its own changes.Java version.
./mvnw verifyon a laptop costs what itdid.
paragraph of
docs/ci.md;masterhas been merged into this branch since, keeping both: eightmodules, and a build of a couple of minutes.
Open questions
None.
masterAGENTS.mdholddocs/DESIGN.mdwithout explicit approval🤖 Generated with Claude Code